1. Field of the Invention
This invention relates to a visual axis detecting device for detecting the visual axis of a user, and more specifically, to a visual axis detecting device for detecting the visual axis of a user by extracting a pupil edge as a boundary between an iris and a pupil.
2. Related Background Art
Conventionally, there have been proposed various types of visual axis detecting devices for detecting where a person looks. For example, the device in Japanese Unexamined Patent Publication No. 1-274736 projects parallel light beams to an eyeball of an observer from a light source and determines a visual axis by making use of an image reflected from a cornea, that is, a so-called cornea reflected image (Purkinje image) and the imaging position of a pupil. An example of this visual axis detecting device will be described with reference to FIG. 12 to FIG. 18.
FIG. 12 is a block diagram showing the main portion of the electrical arrangement of the visual axis detecting device. Numeral 1 denotes a sequence controller for controlling the sequence of a system as a whole, numerals 2 and 3 denote a pair of IREDs disposed in parallel with each other in the direction horizontal to an observer to illuminate an eyeball of the observer, numeral 4 denotes an IRED driver for driving the IREDs 2, 3, numeral 5 denotes an imaging device such as a CCD or the like for reading an eyeball image illuminated by the IREDs 2, 3, numeral 6 denotes an A/D converter for subjecting an image signal read by the imaging device 5 to A/D conversion so that it can be easily processed, numeral 7 denotes a push switch for starting and stopping the detection of a visual axis and numeral 8 denotes a battery for supplying electric power to the entire system.
FIG. 13 is a flowchart showing a series of operations executed by the visual axis detecting device as a first conventional example, FIG. 14 is a view showing an eyeball image projected to the imaging device when the observer approaches the visual axis detecting device and his eyeball is in contact with the device and FIG. 15 is a graph showing an output image of a line where Purkinje images exist on the imaging device 5.
In FIG. 13, the sequence controller 1 first turns on the IREDs 2, 3 toward the eyeball of the observer (step #1). Next, the sequence controller 1 reads an eyeball image at the same time through the imaging device 5 and converts the eyeball image into an image signal by digitalizing it through the A/D converter 6 and stores the image signal to an internal RAM (step #2). Then, the sequence controller 1 extracts a Purkinje image by processing the image signal (step #3).
The Purkinje image is extracted in such a manner that when the image signal has a level higher than a given level and a large inclination, it is regarded as the Purkinje image by making use of the fact that the Purkinje image is bright and has a steep output. For example, it is assumed that an eyeball image is as shown in FIG. 14. In the eyeball image, although the Purkinje images are shown by a and b, since Purkinje image conditions for detecting the Purkinje images are set such that a signal level is L.sub.1 (see FIG. 15) or higher and an inclination is greater than a given level, the Purkinje images a and b satisfy the conditions on a horizontal line y.sub.1 as shown in FIG. 15 and the positions of them can be determined as Ipa and Ipb by the calculation of the center of gravity of the portions exceeding L.sub.1, and the like.
Next, the sequence controller 1 extracts a pupil edge as a boundary between an iris and a pupil by processing the image signal (step #4).
The pupil edge is extracted in such a manner that when the signal has a given level (which is denoted by L.sub.2 as shown in FIG. 15) and an inclination which continues for a predetermined period of time or longer, the intersection point of the signal with the given level L.sub.2 is regarded as the pupil edge by making use of the fact that a pupil portion is dark and an iris portion is brighter than the pupil portion.
For example, when the eyeball image is as shown in FIG. 14, proper pupil edges in the horizontal line y.sub.1 are shown by e.sub.1 and e.sub.6 in FIG. 15. However, since the bottom portions of the Purkinje images a and b also satisfy the conditions of the pupil edge, e.sub.2, e.sub.3, e.sub.4 and e.sub.5 shown, for example, in FIG. 15 are also processed as the pupil edges regardless of whether they are actually the pupil edges. To avoid the above problem, false pupil edges (e.sub.2, e.sub.3, e.sub.4 and e.sub.5 in FIG. 15) located in a certain region in the vicinity of the Purkinje images are removed after the pupil edges are extracted as proposed in Japanese Unexamined Patent Publication No. 6-148509 (step #5).
When the pupil edges are extracted (step #4), they are classified into two groups depending upon the directions of inclination of them when they are viewed from a pupil center, that is, they are classified into left edges which have a negative inclination (e.sub.1 in FIG. 15) and right edges which have a positive inclination (e.sub.6 in FIG. 15).
Although the false pupil edges in the vicinity of the Purkinje images are removed at step #5, false pupil edges are also made by eyelashes and other various types of noises in addition to the above false pupil edges.
The black points in FIG. 14 show all the edges including false pupil edges which can be detected. To remove these false pupil edges, the range in which true pupil edges exist is designated based on Purkinje image position information and the edges located outside of the range are removed as proposed in Japanese Unexamined Patent Publication No. 4-347132 (step #6).
In FIG. 14, F shows the designated range and edges located outside of the range F, for example, the edges e.sub.10, e.sub.11 and the like are removed. Some false edges still remain even if the removing means is employed. To further remove the false edges, the sequence controller 1 statistically processes the remaining false edges and removes the edges which are greatly deviated from the overall edges, as false edges as proposed in Japanese Unexamined Patent Publication No. 6-148509 (step #7).
FIG. 16 is a subroutine showing the "removal by statistic".
In FIG. 16, determined first are the average m.sub.LH and the standard deviation .sigma..sub.LH of the horizontal coordinate of the left edge group (black points shown in FIG. 17) in the remaining edges (step #21). Then, the edges having the coordinates which are greatly deviated from the standard deviation in the left edges, that is, the edges having the coordinates represented by any one of the following formulas are removed (step #22): EQU horizontal coordinate&gt;m.sub.LH +.sigma..sub.LH EQU horizontal coordinate&lt;m.sub.L.sbsb.H -.sigma..sub.LH.
Next, the average m.sub.LV and the standard deviation .sigma..sub.LV in the vertical coordinate of the remaining left edge group are determined (step #23). Then, the edges having the coordinates which are greatly deviated from the standard deviation in the left edge group, that is, the edges having the coordinates represented by any one of the following formulas are removed (step #24). EQU vertical coordinate&gt;m.sub.LV +.sigma.'LV EQU vertical coordinate&lt;m.sub.LV -.sigma..sub.LV
Likewise, the average m.sub.RH and the average deviation .sigma..sub.RH in the horizontal direction of the right edge group (step #25) are determined and the edges having the coordinates which are greatly deviated from the standard deviation in the right edge group, that is, the edges having the coordinates represented by any one of the following formulas are removed (step #26): EQU horizontal coordinate&gt;m.sub.RH +.sigma..sub.RH EQU horizontal coordinate&lt;m.sub.RH -.sigma..sub.RH.
Next, the average m.sub.RV and the standard deviation .sigma..sub.RV in the vertical direction of the remaining right edge group are determined (step #27). Then, the edges having the coordinates which are greatly deviated from the standard deviation in the right edge group, that is, the edges having the coordinates represented by any one of the following formulas are removed (step #29): EQU vertical coordinate&gt;m.sub.RV +.sigma..sub.RV EQU vertical coordinate&lt;m.sub.RV -.sigma..sub.RV.
On the completion of the above processing, a pupil center is detected at step #8 in FIG. 13. The pupil center is determined by the method of least squares or the like by making use of the fact that a pupil portion has a round shape (e.g., a circular shape or an oval shape). After the calculation of the pupil center, a pupil circle estimating error CER is calculated based on the position of the thus determined pupil center and the positions of the respective pupil edges in order to check the certainty of the calculated value (step #9). The calculation is executed using a square error as proposed in Japanese Unexamined Patent Publication No. 4-347131. Then, it is checked whether the determined pupil circle estimating error CER is equal to or less than a threshold value C.sub.T or not (step #10). When it is equal to or less than the threshold value C.sub.T, the rotational angle of an eyeball is calculated based on the position of the determined pupil center and the positions of the Purkinje images (step #12) and the detection of the visual axis is finished.
Whereas, when the pupil circle estimating error CER is greater than the C.sub.T at step #10, since a lot of false pupil edges are contained in the edges, the false pupil edges are further removed, that is, a "correction of the pupil circle" is executed (step #11) and the process returns to the calculation of the pupil center again (step #8).
FIG. 18 is a flowchart explaining the "correction of the pupil circle" at step #11.
In FIG. 18, the sequence controller 1 first checks whether the pupil circle is corrected a first time (first correction) or not (step #31). When it is corrected a first time, the pupil circle estimating error CER is stored as CER.sub.0 (step #33) and one edge is removed (step #35). An order according to which the edges are removed is not particularly taken into consideration here and it is assumed that the left edges are removed first for the time being and the edges in the left edge group are removed in the order they are stored in a memory in the sequence controller 1 (ordinarily, the order they are extracted).
When the correction of the pupil circle at step #301 is not the first time, that is, when the process returns to step #8 in FIG. 13 after the pupil circle is corrected a first time, a second time or thereafter and then executes the "correction of the pupil circle" again at step #11 because the pupil circle estimating error CER is not less than C.sub.T again at step #10, it is checked whether the thus determined latest pupil circle estimating error CER is made smaller than the pupil circle estimating error CER before the edge is removed, that is, smaller than CER.sub.0 (this means that the removed edge is the false pupil edge and the pupil circle is made nearer to a true circle) or not (step #32). Unless "CER&lt;CER.sub.0 " here, the removed edge is returned to the original position because it is not a false pupil edge (step #34) and a next edge is removed in the order the edges are arranged (step #35).
When "CER&lt;CER.sub.0 " at step #32, the removed edge is left as it is and the pupil circle estimating error CER is stored as CER.sub.0 (step #33) and a next edge is removed in the order the edges are arranged (step #35) and the process returns to step #8.
Next, a second conventional example will be described with reference to FIG. 19 to FIG. 21.
FIG. 19 is a flowchart showing a series of the operations executed by the visual axis detecting device as a second conventional example. Note, the electrical arrangement of the visual axis detecting device is the same as that shown in FIG. 12.
In FIG. 19, a sequence controller 1 turns on IREDs 2, 3 toward the eyeball of an observer (step #41). Next, the sequence controller 1 reads an eyeball image at the same time through an imaging device 5 and converts the eyeball image into an image signal by digitalizing it through an A/D converter 6 and stores the image signal to an internal RAM (step #42). Then, the sequence controller 1 extracts a Purkinje image by processing the image signal (step #43).
The Purkinje image is extracted in such a manner that when the image signal has a level higher than a given level and a large inclination, it is regarded as the Purkinje image by making use of the fact that the Purkinje image is bright and has a steep output. For example, it is assumed that an eyeball image is as shown in FIG. 14. In the eyeball image, although the Purkinje images are shown by a and b, since Purkinje image conditions for detecting the Purkinje images are set such that a signal level is L.sub.1 (see FIG. 15) or higher and an inclination is greater than a given level, the Purkinje images a and b satisfy the conditions on a horizontal line y.sub.1 and the positions of them can be determined as Ipa and Ipb by the calculation and the like of the center of gravity of the portions exceeding L.sub.1.
Next, the sequence controller 1 extracts a pupil edge by processing the image signal (step #44).
The pupil edge is extracted in such a manner that when the signal has a given level (which is denoted by L.sub.2 as shown in FIG. 15) and an inclination which continues for a predetermined period of time or longer, the intersection point of the signal with the given level L.sub.2 is regarded as the pupil edge by making use of the fact that a pupil portion is dark and an iris portion is brighter than the pupil portion.
For example, when the eyeball image is as shown in FIG. 14, proper pupil edges in the line y.sub.1 are shown by e.sub.1 and e.sub.6 in FIG. 15. However, since the bottom portions of the Purkinje images a and b also satisfy the condition of the pupil edge, e.sub.2, e.sub.3, e.sub.4 and e.sub.5 are processed as the pupil edges regardless of wether they are the pupil edges. To avoid the above problem, false pupil edges (e.sub.2, e.sub.3, e.sub.4 and e.sub.5 in FIG. 15) located in a certain region in the vicinity of the Purkinje images are removed after the pupil edges are extracted, as proposed in Japanese Unexamined Patent Publication No. 6-148509 (step #45).
False pupil edges are also made by eyelashes and other various types of noises in addition to the false pupil edges in the vicinity of the Purkinje image. The black points in FIG. 17 represent all the edges including the false pupil edges which can be detected as described above. To remove these false pupil edges, the sequence controller 1 limits the range where true pupil edges exist based on Purkinje image position information as proposed in Japanese Unexamined Patent Publication No. 4-347132 (step #46).
FIG. 20 is a subroutine showing a "setting of the designated pupil range".
In FIG. 20, the left coordinate of the coordinates of two Purkinje images is represented by (IP.sub.1, JP.sub.1), the right coordinate of them is represented by (IP.sub.2, JP.sub.2) and the designated pupil range is set as shown below: EQU IS.sub.1 .rarw.IP.sub.1 -20 EQU IS.sub.2 .rarw.IP.sub.2 +20 EQU JS.sub.1 .rarw.(JP.sub.1 +JP.sub.2)/2-40 EQU JS.sub.2 .rarw.(JP.sub.1 +JP.sub.2)/2+20
where, IS.sub.1 represents the left limit of the designated pupil range, IS.sub.2 represents the right limit thereof, JS.sub.1 represents the upper limit thereof, and JS.sub.2 represents the lower limit thereof.
According to the above arrangement, the set range is formed to a rectangular shape on a screen and has a range F to images as shown in FIG. 14.
On the completion of the flowchart in FIG. 20, a "removal of pupil edges outside of the designated pupil range" is executed at step #47 in FIG. 19.
FIG. 21 is a subroutine showing the "removal of pupil edges outside of the designated pupil range".
In FIG. 21, the edges which satisfy "horizontal coordinate&lt;IS.sub.1 " and "horizontal coordinate&gt;IS.sub.2 " are removed from all the edges which have been extracted at step #44 in FIG. 19 and then subjected to the "removal of pupil edges in the vicinity of the Purkinje image" at step #45 (steps #71, #72). Thereafter, the edges which satisfy "vertical coordinate&lt;JS.sub.1 " and "vertical coordinate&gt;JS.sub.2 " are removed from the remaining edges (steps #73, #74).
On the completion of the above processing, a pupil center is detected at step #48 in FIG. 19. The pupil center is determined by the method of least squares or the like by making use of the fact that a pupil portion has a round shape (including an oval shape). After the calculation of the pupil center, a pupil circle estimating error CER is calculated based on the position of the thus determined pupil center and the positions of the respective pupil edges in order to check the certainty of the calculated value (step #50). The calculation is executed using a square error as proposed in Japanese Unexamined Patent Publication No. 4-347131. Then, it is checked whether the determined pupil circle estimating error CER is equal to or less than a threshold value C.sub.T, which is the reference of the certainty, or not (step #51). When it is equal to or less than the threshold value C.sub.T, the rotational angle of an eyeball is calculated based on the position of the determined pupil center and the position of the Purkinje images (step #52) and the detection of the visual axis is finished.
Whereas, when the pupil circle estimating error CER is greater than the C.sub.T at step #50, since a lot of false pupil edges are contained in the edges, the false pupil edges are further removed, that is, the "correction of the pupil circle" (for example, FIG. 18) is executed (step #51) and the process returns to the calculation of the pupil center again (step #48).
A third conventional example will be described with reference to FIG. 24 to FIG. 29.
FIG. 24 is a flowchart showing a series of operations executed by the visual axis detecting device as the third example, FIG. 25 is a view of an eyeball image projected to the imaging device 5 in FIG. 12 when an observer approaches the visual axis detecting device and his eyeball is in contact with the device and FIG. 26 is a graph showing an output image of a line where Purkinje images exist on the imaging device 5.
In FIG. 24, a sequence controller 1 first turns on IREDs 2, 3 toward the eyeball of the observer (step #A1). Next, the sequence controller 1 reads an eyeball image at the same time through the imaging device 5 and converts the eyeball image into an image signal by digitalizing it through an A/D converter 6 and stores the image signal to an internal RAM (step #A2). Then, the sequence controller 1 extracts a Purkinje image by processing the image signal (step #A3).
The Purkinje image is extracted in such a manner that when the image signal has a level higher than a given level and a large inclination, it is regarded as the Purkinje image by making use of the fact that the Purkinje image is bright and has a steep output. For example, it is assumed that an eyeball image is as shown in FIG. 25. In the eyeball image, although the Purkinje images are shown by a and b, since Purkinje image conditions for detecting the Purkinje images are set such that a signal level is T.sub.P or higher and an inclination is greater than a given level, the Purkinje images a and b satisfy the conditions on a horizontal line y as shown in FIG. 26 and the positions of them can be determined as Xa and Xb by the calculation of the center of gravity of the portions exceeding T.sub.P, and the like.
Next, the sequence controller 1 extracts a pupil edge by processing the image signal (step #A4).
The pupil edge is extracted in such a manner that when the signal has a given level (which is denoted by T.sub.D as shown in FIG. 26) and an inclination which continues for a predetermined period of time or longer, the intersection point of the signal with the given level T.sub.D is regarded as the pupil edge by making use of the fact that a pupil portion (portion c in FIG. 25) is dark and an iris portion (portion d in FIG. 25) is brighter than the pupil portion.
For example, when the eyeball image is as shown in FIG. 25, proper pupil edges in a horizontal line y are shown by e.sub.1 and e.sub.6 in FIG. 26. However, since the bottom portions of the Purkinje images a and b also satisfy the condition of the pupil edge, e.sub.2, e.sub.3, e.sub.4 and e.sub.5 are also processed as the pupil edges regardless of whether they are the pupil edges. Therefore, when a pupil center is determined by the method of least squares or the like using all the pupil edges, the pupil center is displaced by the false pupil edges and, as a result, the rotational angle of the eyeball of an observer which is determined finally is displaced.
Therefore, the sequence controller 1 previously removes the edges which are regarded as the false pupil edges within a certain region in the vicinity of the Purkinje image after the removal of the pupil edges in order to remove the false pupil edges at the bottom portion of the Purkinje image as proposed in Japanese Unexamined Patent Publication No. 6-148509 (step #A5).
An example of the processing at step #A5, that is, a processing for removing the false pupil edges within the certain region in the vicinity of the Purkinje image will be described with reference to the flowchart in FIG. 27.
When the coordinates of two Purkinje images a and b which have been extracted are represented by (Xa, Yb) and (Xb, Yb), respectively, the sequence controller 1 sets the width of a pupil edge removing region to k (tentatively, k=2) to both the coordinates in both the X-direction (horizontal direction) and the Y-direction (vertical direction), respectively (step #A501). Next, as the removal of the bottom portion of the Purkinje image a, the sequence controller 1 removes the edges in the respective regions equal to or greater than (XA-k) and equal to or less than (Xa+k) in the horizontal direction and equal to or greater than (Ya-k) and equal to or less than (Ya+k) in the vertical direction (step #A502). With this processing, the false pupil edges e.sub.2 and e.sub.3 shown in FIG. 26 are removed.
Next, the sequence controller 1 removes the false pupil edges within the respective regions equal to or greater than (Xb-k) and equal to or less than (Xb+k) in the horizontal direction and equal to or greater than (Yb-k) and equal to or less than (Yb+k) in the vertical direction as the removal of the bottom portion of the Purkinje image b (step #A503). With this processing, the false pupil edges e.sub.4 and e.sub.5 shown in FIG. 26 are removed.
Returning to FIG. 24, next, the sequence controller 1 determines the pupil center by the method of least squares or the like (step #A6) and the rotational angles of eyeballs .theta..sub.x, .theta..sub.y by the pupil center and the Purkinje images a, b (step #A7), finally decides whether the derived values are correct or not (step #8) and finishes the detection of the visual axis.